Semi-supervised Regression with Order Preferences

Following a discussion on the general form of regularization for semi-supervised learning, we propose a semi-supervised regression algorithm. It is based on the assumption that we have certain order preferences on unlabeled data (e.g., point x1 has a larger target value than x2). Semi-supervised learning consists of enforcing the order preferences as regularization in a risk minimization framework. The optimization problem can be effectively solved by a linear program. Experiments show that the proposed semi-supervised regression outperforms standard regression. 1 Semi-supervised learning as regularization on unlabeled data Semi-supervised learning works when its assumption on unlabeled data, often expressed as regularization, fits the reality of the problem domain. In this paper we first generalize the regularization formulation of some common semi-supervised learning approaches, namely manifold regularization, semi-supervised support vector machines, and multi-view learning [1, 2, 3]. Regularization for each individual approach is not new. However these approaches have been studied largely in isolation. Our general form serves as a bridge to connect them, and to inspire novel semi-supervised approaches. As an example of the latter, we propose a novel algorithm for semi-supervised regression. The proposed regression algorithm is able to incorporate domain knowledge about the relative order of target values on unlabeled points. It thus differs from, and complements, existing semi-supervised regression methods, which do not use such domain knowledge but require multiple views [4, 5]. Let us review the three common semi-supervised learning methods. Manifold regularization [6, 7] generalizes several graph-based semi-supervised learning methods. Let l be the number of labeled points, u the number of unlabeled points. Graph-based semi-supervised learning requires a weighted, undirected graph, characterized by an (l + u) × (l + u) weight matrix W defined on labeled and unlabeled data. It is assumed that from the features of two points xi, xj (e.g., by computing their Euclidean distance), domain experts can assign a non-negative weight wij . A large wij implies a preference for f(xi), f(xj) to be similar. Therefore subgraphs with large weights tend to have the same label. This is sometimes called the cluster assumption. Let K be a kernel and H the corresponding Reproducing Kernel Hilbert Space (RKHS). Let y be the labels, which can be categories for classification or real numbers for regression. Manifold regularization seeks a prediction function f ∈ H, such that f is the solution to